Head mounted display device, image display system, and method of controlling head mounted display device

ABSTRACT

A head mounted display device includes an image display unit that allows a user to visually recognize image light based on image data as a virtual image and allows the user to visually recognize an outside scenery when worn on a head of the user, an imaging unit that images the outside scenery, and a control unit that, when an imaged image contains a mark image as an image of a specific mark, allows the user to visually recognize the specific virtual image associated with a combination of a type of the mark image and a shape of the mark image using the image display unit.

This application is a continuation application of U.S. patentapplication Ser. No. 14/564,635, filed on Dec. 9, 2014, which claims thebenefit of foreign priority based on Japanese patent application Ser.No. 2013-269065, filed Dec. 26, 2013. The disclosures of U.S. patentapplication Ser. No. 14/564,635, filed Dec. 9, 2014, and Japanese PatentApplication No. 2013-269065, filed Dec. 26, 2013, are expresslyincorporated by reference herein in their entirety.

BACKGROUND

1. Technical Field

The present invention relates to a head mounted display device.

2. Related Art

Head mounted display devices (head mounted displays, HMDs) as displaydevices worn on heads have been known. For example, the head mounteddisplay device generates image light representing an image using aliquid crystal display and a light source, guides the generated imagelight to an eye of a user using a projection system and a light guideplate, and thereby, allows the user to visually recognize a virtualimage. Further, in a picking system using barcodes or the like, a useris allowed to visually recognize information using a virtual image bythe head mounted display device, and thereby, a picking operation isstreamlined.

Patent Document 1 (JP-A-10-147411) discloses a technology, when a userwearing a head mounted display device performs a picking operation ofacquiring products stored on the shelves, of allowing the user tovisually recognize images of products corresponding to shelf numbers inaddition to the shelf numbers of the shelves visually recognized by theuser. Further, Patent Document 2 (JP-A-2008-15634) discloses atechnology, in a system of providing information related to products,when a user visually recognizes a predetermined product name in a listof a plurality of product names for a certain period of time or more, ofallowing the user to visually recognize information related to theproduct name visually recognized as an image. Furthermore, PatentDocument 3 (Japanese Patent No. 5334145) discloses a support system forpicking operation, when a plurality of codes placed in a storagelocation of products are imaged by a camera that performs imaging in aline-of-sight direction of a user, of allowing the user to visuallyrecognize an image based on one code selected from the plurality ofcodes based on the information stored in advance.

However, in the technology disclosed in Patent Document 1, there is aproblem that, when the shelf numbers attached to the shelves are smallerand the user and the shelf numbers are apart, It is impossible to allowthe user to visually recognize the images of the products because theshelf numbers are not recognized. Further, there is a problem that it isimpossible to perform an efficient picking operation only by the productimages, In the technology disclosed in Patent Document 2, there is aproblem that, when the user desires to visually recognize an imagerelated to a product name, it is necessary to visually recognize theproduct name on the list for the predetermined period of time or moreand take time to visually recognize the image. In the technologydisclosed in Patent Document 3, for the user to visually recognize animage based on a code, it necessary to register information in advanceand there is room for improvement in user-friendliness. The problems arenot limited to the field of the picking system, but common in otherfields including e.g., route guidance to a destination or the like.

SUMMARY

An advantage of some aspects of the invention is to solve at least apart of the problems described above and the invention can beimplemented as the following aspects.

(1) An aspect of the invention provides a head mounted display device.The head mounted display device includes an image display unit thatallows a user to visually recognize image light based on image data as avirtual image and allows the user to visually recognize an outsidescenery when worn on a head of the user, an imaging unit that images theoutside scenery, and a control unit that, when an imaged age contains amark image as an image of a specific mark, allows the user to visuallyrecognize the specific virtual image associated with a combination of atype of the mark image and a shape of the mark image using the imagedisplay unit. According to the head mounted display device of theaspect, the specific virtual image corresponding to a positionalrelationship between the user and the imaged specific mark specified inresponse to the shape of the imaged mark image, and thereby, the usermay virtually recognize the optimal image in response to the type andthe positional relationship of the imaged specific mark and mayefficiently perform various operations based on the visually recognizedvirtual images.

(2) In the head mounted display device of the aspect described above,the specific virtual image may be associated with a combination of thetype of the mark image and a size of the mark image as the shape of themark image. According to the head mounted display device of the aspect,even when the same specific mark is imaged, a plurality of specificvirtual images may be visually recognized by the user in response to thespecified distances, and thereby, even one specific marker may providemore information suitable for the positional relationship with the userto the user.

(3) In the head mounted display device of the aspect described above,the control unit may set a distance from the image display unit to alocation where the specific virtual image is formed to be equal to adistance from the image display unit to the specific mark specifiedbased on the combination of the type of the mark image and the size ofthe mark image. According to the head mounted display device of theaspect, the specific virtual image may be formed in the location wherethe user easily and visually recognizes the virtual image, and thereby,the specific virtual image may be promptly and visually recognized bythe user.

(4) In the head mounted display device of the aspect described above,the specific virtual image may be associated with a combination of anorder of imaging of the one mark image contained in each of a pluralityof imaged images, the type of the mark image, and the shape of the markimage. According to the head mounted display device of the aspect, ofthe plurality of mark images, only the specific virtual imagecorresponding to the specific mark along the order of imaging isvisually recognized by the user, and thereby, virtual images related tovarious operations in the different order may not be visually recognizedby the user and efficient operations may be performed by the user.

(5) In the head mounted display device of the aspect described above,the control unit may allow the user to visually recognize an areavirtual image corresponding to a predetermined area in the imaged imageusing the image display unit, and allow the user to visually recognizethe specific virtual image associated with the combination of the markimage contained in the predetermined area using the image display unit.According to the head mounted display device of the aspect, a specificmark only contained in the imaged image is not detected as a mark imagefor the user to visually recognize a specific virtual image, and theload of the processing on the head mounted display device may bereduced. Further, detection of mark images not intended by the user maybe suppressed and specific virtual images corresponding to variousoperations with higher accuracy may be visually recognized by the user.

(6) In the head mounted display device of the aspect described above,when the imaged image contains a plurality of the mark images, thecontrol unit may allow the user to visually recognize the specificvirtual image associated with the combination of the mark image havingthe maximum size of the plurality of the mark images using the imagedisplay unit. According to the head mounted display device of theaspect, the maximum mark image is automatically detected from theplurality of mark images, and thereby, the specific marker closest tothe user may be automatically detected and convenience of the user maybe improved.

(7) In the head mounted display device of the aspect described above,when the imaged image contains a plurality of the mark images, thecontrol unit may allow the user to visually recognize the specificvirtual image associated with the combination of the one mark imageselected based on priority set for the specific mark using the imagedisplay unit. According to the head mounted display device of theaspect, of the plurality of mark images contained in the imaging range,the specific virtual image corresponding to the mark image stored inadvance may be automatically and visually recognized by the user, andthereby, visual recognition by the user of erroneous specific virtualimages of various operations may be reduced and efficient operations maybe performed by the user.

(8) In the head mounted display device of the aspect described above,the specific virtual image may be associated with a combination of atime when the imaged image contains the mark image, the type of the markimage, and the shape of the mark image. According to the head mounteddisplay device of the aspect, a specific mark only contained in theimaged image is not detected as a mark image for the user to visuallyrecognize a specific virtual image, and the load of the processing onthe head mounted display device may be reduced. Further, detection ofmark images not intended by the user may be suppressed and specificvirtual images corresponding to various operations with higher accuracymay be visually recognized by the user.

(9) In the head mounted display device of the aspect described above,the specific mark may be formed by a combination of a shape and colors.According to the head mounted display device of the aspect, the specificmark using a plurality of colors is created, and thereby, even thesmaller specific mark may provide more information to the user.

(10) In the head mounted display device of the aspect described above,the specific mark may be a two-dimensional mark having an outer frame ofa square in which a plurality of areas are in different colors of whiteand black. According to the head mounted display device of the aspect,the created specific mark is not complex, and thereby, control fordisplaying a corresponding image by the imaged mark image may beperformed more easily. Further, the specific mark is formed in black andwhite only, and thereby, control with less erroneous detection may beperformed.

Not all of the plurality of component elements of the respective aspectsof the invention described above are essential. In order to solve partor all of the above described problems or in order to achieve part orall of the advantages described in the specification, some componentelements of the plurality of the component elements may be appropriatelychanged, deleted, replaced by new other component elements, or partiallydeleted in limitations. Further, in order to solve part or all of theabove described problems or in order to achieve part or all of theadvantages described in the specification, part or all of the technicalfeatures contained in one of the aspects of the invention describedabove may be combined with part or all of the technical featurescontained in the other one of the aspects of the invention describedabove into one independent aspect of the invention.

For example, one aspect of the invention may be implemented as a deviceincluding one, two, or more of the three elements of the image displayunit, the imaging unit, and the control unit. That is, the device mayhave the image display unit or not. Or, the device may have the imagingunit or not. Or, the device may have the control unit or not. The imagedisplay unit may allow a user to visually recognize image light as avirtual image based on image data and allow the user to visuallyrecognize an outside scenery when worn on a head of the user, forexample. The imaging unit may image the outside scenery, for example.When an imaged image contains a mark image as an image of a specificmark, the control unit may allow the user to visually recognize thespecific virtual image corresponding to a combination of a type of themark image and a shape of the mark image using the image display unit,for example. The device may be implemented as a head mounted displaydevice, for example, and may be implemented as other devices than thehead mounted display device. According to the aspect, at least one ofvarious challenges including improvement and simplification inoperability of the device, integration of the device, improvement inconvenience of the user using the device may be resolved. Any of part orall of the technical features of the above described respective aspectsof the head mounted display device may be applied to the device.

The invention may be implemented in other various aspects than the headmounted display device. For example, the invention may be implemented informs of a method of controlling the head mounted display device, animage display system and a head mounted display system, a computerprogram for realization of the head mounted display system and the imagedisplay system, a recording medium recording the computer program, datasignals embodied within carrier wave containing the computer program,etc.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the accompanyingdrawings, wherein like numbers reference like elements.

FIG. 1 is an explanatory diagram showing an outer configuration of ahead mounted display device.

FIG. 2 is a block diagram functionally showing a configuration of thehead mounted display device.

FIG. 3 is an explanatory diagram showing image lights output by an imagelight generation part.

FIG. 4 is an explanatory diagram showing a flow of instruction imagedisplay processing.

FIG. 5 is an explanatory diagram showing an example of a marker.

FIG. 6 is a schematic diagram with respect to a path that a user takesin a distribution center.

FIG. 7 is an explanatory diagram showing a visual range of the user whena marker is detected at a start point.

FIG. 8 is an explanatory diagram showing a visual range of the user whenan instruction image is displayed at the start point.

FIG. 9 is an explanatory diagram showing a visual range of the user whenan instruction image is displayed at a relay point.

FIG. 10 is an explanatory diagram showing a visual range of the user infront of a shelf.

FIG. 11 is an explanatory diagram showing an example of the marker in amodified example.

FIG. 12 is an explanatory diagram showing an example of the marker in amodified example.

FIG. 13 is an explanatory diagram showing an example of the marker in amodified example.

FIGS. 14A and 14B are explanatory diagrams showing outer configurationsof head mounted display devices in modified examples.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

A. Embodiment:

A-1. Configuration of Head Mounted Display Device:

FIG. 1 is an explanatory diagram showing an outer configuration of ahead mounted display device 100. The head mounted display device 100 adisplay device worn on a head and also called a head mounted display(HMD) The head mounted display device 100 of the embodiment is anoptically-transmissive head mounted display device that enables visualrecognition of a virtual image and direct visual recognition of anoutside scenery. Note that, in the specification, the virtual imagevisually recognized by the user using the head mounted display device100 is also referred to as “displayed image” for convenience. Further,output of image light generated based on image data is also referred toas “display of image”.

The head mounted display device 100 includes an image display unit 20which allows the user to visually recognize a virtual image when worn onthe head of the user, and a control unit 10 (controller 10) thatcontrols the image display unit 20.

The image display unit 20 is a wearable unit worn on the head of theuser and has a spectacle shape in the embodiment The image display unit20 includes a right holding part 21, a right display drive part 22, aleft holding part 23, a left display drive part 24, a right opticalimage display part 26, a left optical image display part 28, and acamera 61. The right optical image display part 26 and the left opticalimage display part 28 are provided to be located in front of the rightand left eyes of the user when the user wears the image display unitrespectively. One end of the right optical image display part 26 and oneend of the left optical image display part 28 are connected to eachother in a location corresponding to the glabella of the user when theuser wears the image display unit 20.

The right holding part 21 is a member provided to extend from an endpart ER as the other end of the right optical image display part 26 tothe location corresponding to the temporal part of the user when theuser wears the image display unit 20. Similarly, the left holding part23 is a member provided to extend from an end part EL as the other endof the left optical image display part 28 to the location correspondingto the temporal part of the user when the user wears the image displayunit 20. The right holding part 21 and the left holding part 23 hold theimage display unit 20 on the head of the user like temples ofspectacles.

The right display drive part 22 and the left display drive part 24 areprovided at the sides opposed to the head of the user when the userwears the image display unit 20. Note that, as below, the right holdingpart 21 and the left holding part 23 are also collectively and simplyreferred to as “holding parts”, the right display drive part 22 and theleft display drive part 24 are also collectively and simply referred toas “display drive parts”, and the right optical image display part 26and the left optical image display part 28 are also collectively andsimply referred to as “optical image display parts”.

The display drive parts 22, 24 include liquid crystal displays 241, 242(hereinafter, also referred to as “LCDs 241, 242”), projection systems251, 252, and the like (see FIG. 2) The details of the configurations ofthe display drive parts 22, 24 will be described later. The opticalimage display parts 26, 28 as optical members include light guide plates261, 262 (see FIG. 2) and a dimming plate. The light guide plates 261,262 are formed using a light-transmissive resin material or the like andguide image lights output from the display drive parts 22, 24 to theeyes of the user, The dimming plate is an optical device having a thinplate shape and provided to cover the front side of the image displayunit as the opposite side to the sides of the eyes of the user. Thedimming plate protects the light guide plates 261, 262 and suppressesdamage, attachment of dirt, or the like to the light guide plates 261,262. Further, by adjustment of light transmittance of the dimming plate,the amount of outside light entering the eyes of the user may beadjusted and the ease of visual recognition of the virtual image may beadjusted. Note that the dimming plate is dispensable.

A camera 61 is provided in a location corresponding to the glabella ofthe user when the user wears the image display unit 20. Accordingly, thecamera 61 images an outside scenery as a scenery outside in aline-of-sight direction of the user and acquires an imaged image whenthe user wears the image display unit 20 on the head. The camera 61 is amonocular camera, or may be a stereo camera. The camera 61 correspondsto an imaging unit in the appended claims.

The image display unit 20 further has a connection unit 40 forconnecting the image display unit 20 to the control unit 10. Theconnection unit 40 includes a main body cord 48 connected to the controlunit 10, a right cord 42, a left cord 44, and a coupling member 46. Theright cord 42 and the left cord 44 are cords bifurcated from the mainbody cord 48. The right cord 42 is inserted into a casing of the rightholding part 21 from an end part AP in the extension direction of theright holding part 21 and connected to the right display drive part 22.Similarly, the left cord 44 is inserted into a casing of the leftholding part 23 from an end part AP in the extension direction of theleft holding part 23 and connected to the left display drive part 24.The coupling member 46 is provided at the bifurcation point of the mainbody cord 48 and the right cord 42 and the left cord 44, and has a jackfor connection of an earphone plug 30. From the earphone plug 30, aright earphone 32 and a left earphone 34 extend.

The image display unit 20 and the control unit 10 perform transmissionof various signals via the connection unit 40. Connectors (not shown)fitted in each other are respectively provided in the end part in themain body cord 48 opposite to the coupling member 46 and the controlunit 10. By fit/unfit of the connector of the main body cord 48 and theconnector of the control unit 10, the control unit 10 and the imagedisplay unit 20 are connected or disconnected, For example, metal cablesand optical fibers may be employed for the right cord 42, the left cord44, and the main body cord 48.

The control unit 10 is a device for controlling the head mounted displaydevice 100. The control unit 10 includes an enter key 11, a lightingpart 12, a display change key 13, a brightness change key 15, an arrowkey 16, a menu key 17, a power switch 18, and a track pad 14. The enterkey 11 detects a press operation and outputs a signal for deciding thecontents rated in the control unit 10. The lighting part 12 notifies theuser of the operation status of the head mounted display device 100 byits emission state. The operation status of the head mounted displaydevice 100 includes ON/OFF of power, for example. As the lighting part12, for example, an LED (Light Emitting Diode) is used. The displaychange key 13 detects a press operation and outputs a signal forswitching the display mode of content video between 3D and 2D, forexample. The brightness change key 15 detects a press operation andoutputs a signal for increasing and decreasing the brightness of theimage display unit 20. The arrow key 16 detects a press operation forthe key corresponding to up, down, right and left and outputs a signalin response to the detected operation. The power switch 18 detects aslide operation of the switch, and thereby, switches the power-on stateof the head mounted display device 100.

The track pad 14 detects a touch operation of a finger by the user onthe operation surface and outputs a signal in response to the detectedoperation. As the track pad 14, various track pads of electrostatictype, pressure detection type, and optical type may be employed.

FIG. 2 is a block diagram functionally showing a configuration of thehead mounted display device 100. As shown in FIG. 2, the control unit 10has an operation part 135, a power source 130, a memory part 120, a CPU140, an interface 180, a transmission part 51 (Tx 51), and atransmission part 52 (Tx 52). The operation part 135 receives operationsby the user and includes the enter key 11, the display change key 13,the track pad 14, the brightness change key 15, the arrow key 16, themenu key 17, and the power switch 18.

The power source 130 supplies power to the respective units of the headmounted display device 100. As the power source 130, for example, asecondary cell may be used. The memory part 120 stores various computerprograms. The memory part 120 includes a ROM, a RAM, or the like. Thememory part 120 stores markers detected by a marker determination part165, which will be described later, and image data representing imagescorresponding to the respective markers and displayed on the imagedisplay unit 20 in advance. The memory part 120 stores markers relatedto operation instructions of instruction image display processing, whichwill be described later. The CPU 140 loads and executes the computerprograms stored in the memory part 120, and thereby, functions as anoperating system 150 (OS 150), a display control part 190, an imageprocessing part 160, a sound processing part 170, and the markerdetermination part 165.

The display control part 190 generates control signals for controllingthe right display drive part 22 and the left display drive part 24.Specifically, the display control part 190 individually controls driveON/OFF of the right LCD 241 by a right LCD control part 211, driveON/OFF of a right backlight 221 by a right backlight control part 201,drive ON/OFF of the left LCD 242 by a left LCD control part 212 driveON/OFF of a left backlight 222 by a left backlight control part 202,etc. with the control signals. Thereby, the display control part 190controls the respective generation and output of mage lights by theright display drive part 22 and the left display drive part 24. Forexample, the display control part 190 may allow both the right displaydrive part 22 and the left display drive part 24 to generate imagelights, allow only one of the parts to generate image light, or allowneither to generate image lights. The display control part 190 transmitsthe respective control signals for the right LCD control part 211 andthe left LCD control part 212 via the transmission parts 51 and 52.Further, the display control part 190 transmits the respective controlsignals for the right backlight control part 201 and the left backlightcontrol part 202.

The image processing part 160 acquires image signals contained incontents. The image processing part 160 separates synchronizing signalsincluding a vertical synchronizing signal VSync and a horizontalsynchronizing signal HSync from the acquired image signals. Further, theimage processing part 150 generates clock signals PCLK using a PLL(Phase Locked Loop) circuit or the like (not shown) in response to theperiods of the separated vertical synchronizing signal VSync andhorizontal synchronizing signal HSync. The image processing part 160converts the analog image signals from which the synchronizing signalshave been separated into digital image signals using an A/D convertercircuit or the like (not shown). Then, the image processing part 160stores the converted digital image signals as image data (RGB data) ofan object image in a DRAM within the memory part 120 with respect toeach frame. Note that the image processing part 160 may execute imageprocessing such as resolution conversion processing, various kinds oftone correction processing including adjustment of brightness andsaturation, keystone correction processing, or the like on the imagedata as necessary.

The image processing part 160 transmits the respective generated clocksignals PCLK, vertical synchronizing signal VSync, horizontalsynchronizing signal HSync, and the image data stored in the DRAM withinthe memory part 120 via the transmission parts 51, 52. Note that theimage data transmitted via the transmission part 51 is also referred toas “right-eye image data” and the image data transmitted via thetransmission part 52 is also referred to as “left-eye image data”. Thetransmission parts 51, 52 function as transceivers for serialtransmission between the control unit 10 and the image display unit 20.

The marker determination part 165 displays an area image showing acertain area in an image display maximum range PN (hereinafter, simplyreferred to as “display range PN”) that may be visually recognized bythe user in the optical image display parts 26, 28 of the image displayunit 20. Note that the area image corresponds to an area virtual imagein the appended claims. The marker determination part 165 determineswhether or not there is an image of a marker stored in the memory part120 in a range corresponding to the area image in the imaged image ofthe camera 61. If the imaged image contains the image of the marker, butthe image of the whole marker is not contained within the area image,the marker determination part 165 determines that the imaged image doesnot contain the marker. Further, if determining that the marker iscontained within the area image, the marker determination part 165determines whether or not the imaged marker is a marker related to anoperation instruction of the instruction image display processing to bedescribed later by checking the marker against the markers stored in thememory part 120. The marker is an identifier including a barcode and aQR code (registered trademark) one-dimensionally or two-dimensionallyrepresented. As the marker, e.g., an AR marker, an ID marker, an NyIDmarker, a DataMatrix marker, a frame marker, a split marker, markershaving increased amounts of information formed by addition of colors tothe identifiers, a marker created by a multidimensional code, acharacter readable by an OCR or the like are used. The markercorresponds to a specific mark in the appended claims.

if determining that the imaged image contains the image of the marker inthe range of the area image, the marker determination part 165 specifiesthe type of the marker contained in the imaged image and specifies theshape of the image of the marker. Even when the imaged markers are ofthe same type, the images of the markers may be different in image sizeor the shapes of the outer frames of the images may be differentdepending on the positional relationship with the camera 61. Forexample, in the case where the distance between the imaged marker andthe camera 61 is longer, the size of the image of the imaged marker issmaller than that in the case where the distance is not longer. Further,in the case where the imaged marker is located diagonally ahead of thecamera 61, the marker is imaged as an image with a depth, and, when theouter frame of the marker is square, for example, the outer frame of theimage of the marker is imaged as a trapezoidal image. As describedabove, the shape of the image of the marker varies depending on thepositional relationship between the imaged marker and the camera 61, andthe marker determination part 165 may specify the distance between theuser and the imaged marker, the location of the imaged marker withrespect to the line-of-sight direction of the user, or the like usingthe age of the marker. The marker determination part 165 displays animage corresponding to a combination of the type of the specified markerand the shape of the image of the marker in the display range PN. Theuser may obtain information from the image displayed in the displayrange PN and make various judgments. The image of the imaged markercorresponds to a mark image in the appended claims.

Further, the marker determination part 165 controls an angle ofconvergence, and thereby, sets the location where the virtual imagevisually recognized by the user is formed. That is, the markerdetermination part 165 allows the virtual image visually recognized bythe user to be visually recognized as if the image exists in a locationat a predetermined distance from the user. In the embodiment, the markerdetermination part 165 sets the distance from the image display unit 20to the location where the virtual image is formed to be equal to thedistance from the camera 61 to the imaged marker. Note that, in theappended claims, the description that a distance from an image displayunit to a location where a specific virtual image is formed is equal toa distance from the image display unit to a specific mark is not limitedto that the respective distances are completely the same, but includesthe distance that falls within ±10% around a center value as thedistance between the imaging unit and the specific mark. For example,when the distance from the camera 61 to the imaged marker is five meters(m), if the distances from the optical image display parts 25, 28 to thelocation where the virtual image is formed are from 4.5 m to 5.5 m, thedistances correspond to the same distance in the appended claims.

The sound processing part 170 acquires sound signals contained in thecontents, amplifies the acquired sound signals, and supplies the signalsto a speaker (not shown) within the right earphone 33 and a speaker (notshown) within the left earphone 34 connected to the coupling member 46.Note that, for example, in the case where the Dolby (registeredtrademark) system is employed, processing on the sound signals isperformed and different sounds at the varied frequencies or the like areoutput from the respective right earphone 32 and left earphone 34.

The interface 180 (FIG. 2) is an interface for connecting variousexternal devices OA as supply sources of contents to the control unit10. The external devices OA include a personal computer (PC), a cellphone terminal, a game terminal, etc., for example. As the interface180, for example, a USE interface, a micro USE interface, an interfacefor memory card, or the like may be used.

The image display unit 20 includes the right display drive part 22, theleft display drive part 24, the right light guide plate 261 as the rightoptical image display part 26, and the left light guide plate 262 as theleft optical image display part 28.

The right display drive part 22 includes a reception part 53 (Rx 53),the right backlight control part 201 (right BL control part 201) and theright backlight 221 (right BL 221) that function as a light source, theright LCD control part 211 and the right LCD 241 that function as adisplay device, and the right protection system 51. The right backlightcontrol part 201 and the right backlight 221 function as the lightsource. The right LCD control part 211 and the right LCD 241 function asthe display device. Note that the right backlight control part 201, theright LCD control part 211, right backlight 221, and the right LCD 241are also collectively referred to as “image light generation part”.

The reception part 53 functions as a receiver for serial transmissionbetween the control unit 10 and the image display unit 20. The rightbacklight control part 201 drives the right backlight 221 based on theinput control signal. The right backlight 221 is a light emitter such asan LED or electroluminescence (EL), for example. The right LCD controlpart 211 drives the right LCD 241 based on the clock signal PCLK, thevertical synchronizing signal VSync, the horizontal synchronizing signalHSync, and the right-eye image data input via the reception part 53. Theright LCD 241 is a transmissive liquid crystal panel in which aplurality of pixels are arranged in a matrix.

The right projection system 251 includes a collimator lens that bringsthe image light output from the right LCD 241 into parallelized luminousfluxes. The right light guide plate 261 as the right optical imagedisplay part 26 guides the image light output from the right projectionsystem 251 to the right eye RE of the user while reflecting the lightalong a predetermined optical path. Note that the right projectionsystem 251 and the right light guide plate 261 are also collectivelyreferred to as “light guide part”.

The left display drive part 24 has the similar configuration as that ofthe right display drive part 22. The left display drive part 24 includesa reception part 54 (Rx 54), the left backlight control part 202 (leftBL control part 202) and the left backlight 222 (left BL 222) thatfunction as a light source, the left LCD control part 212 and the leftLCD 242 that function as a display device, and the left projectionsystem 252. The left backlight control part 202 and the left backlight222 function as the light source. The left LCD control part 212 and theleft LCD 242 function as the display device. Note that the leftbacklight control part 202, the left LCD control part 212, the leftbacklight 222, and the left LCD 242 are also collectively referred to as“image light generation part”. Further, the left projection system 252includes a collimator lens that brings the image light output from theleft LCD 242 into parallelized luminous fluxes. The left light guideplate 262 as the left optical image display part 28 guides the imagelight output from the left projection system 252 to the left eye LE ofthe user while reflecting the light along a predetermined optical path.Note that the left projection system 252 and the left light guide plate262 are also collectively referred to as “light guide part”.

FIG. 3 is an explanatory diagram showing image lights output by theimage light generation part The right LCD 241 drives the liquid crystalin the respective pixel positions arranged in the matrix to change thetransmittance of the light to be transmitted through the right LCD 241,and thereby, modulates illumination light IL radiated from the rightbacklight 221 into effective image light PL representing an image. Thisapplies to the left side. Note that the backlight system is employed inthe embodiment as shown in FIG. 3, however, a configuration that outputsimage light using the front light system or the reflection system may beemployed.

A-2. Instruction Image Display Processing:

FIG. 4 is an explanatory diagram showing a flow of instruction imagedisplay processing. The instruction image display processing isprocessing of sequentially displaying instruction images showingoperation instructions in the display range PN of the image display unit20 along the preset operation instructions in response to the detectedmarkers. In the embodiment, a plurality of different markers placed inthe distribution center in which a large number of industrial productsare stored are detected, and thereby, images of operation instructionscorresponding to the images of the markers are displayed in the displayrange PN. FIG. 5 is an explanatory diagram showing an example of amarker. As shown in FIG. 5, in markers MK1 used in the embodiment,different images are formed with respect to each marker in square outerframes. In the marker MK1, the age formed in the square includes a whitearea and a black area. Note that the sizes of the outer frames in imagedata of the markers stored in the memory part 120 differ marker tomarker.

FIG. 6 is a schematic diagram with respect to a path that a user takesin a distribution center 300. FIG. 6 schematically shows the entire ofthe distribution center 300. First, the user is at a start point ST nearthe entrance of the distribution center 300 and faces in a direction DR1within the distribution center 300. When the markers placed in therespective locations of the distribution center 300 are detected, theuser moves forward in direction of arrows shown in FIG. 6 according tothe instruction images displayed in response to the images of thedetected markers, and moves to a shelf SH as a destination. After movingto the shelf SH, the user acquires a component stored on the shelf SH asmemorized as the operation instruction in advance. The details of arelay point LC and a direction DR2 shown in FIG. 6 will be describedlater.

In the instruction image display processing, first, the camera 61 Startsimaging of an outside scenery (step S10 in FIG. 4). Then, the markerdetermination part 165 displays an area image in the display range PN(step S12). Then, the marker determination part 165 monitors detectionof an image of a marker within the area image (step S14). If any imageof a marker is not detected within the area image (step S14: NO), themarker determination part 165 continues to monitor detection of an imageof a marker (step S14). In the embodiment, if a marker is continuouslyimaged within the area image a predetermined period of time, the markerdetermination part 165 detects the marker as a marker. Accordingly, amarker instantaneously contained within the area image is not detectedas a marker. In the processing at step S14, if an image of a marker isdetected within the area image (step S14: YES), the marker determinationpart 165 specifies a type of the detected marker (step S16).

FIG. 7 an explanatory diagram showing a visual range VR of the user whena marker is detected at the start point ST. FIG. 7 shows the visualrange VR visually recognized by the user when the user at the startpoint ST faces in the direction DR1. In this case, the visual range VRof the user contains an outside scenery SC through the optical imagedisplay parts 26, 28 of the image display unit 20 and an area image Aldisplayed by the image display unit 20. The outside scenery SC containsthe marker MK1 placed within the distribution center 300. The marker MK1as a marker is detected within the area image AI, and the markerdetermination part 165 specifies the type of the marker of the markerMK1. Note that the image display unit 20 may display an image in a rangeinside of the display range PN shown in FIG. 5. In FIG. 5, forconvenience, the range showing the display range PN is shown by a brokenline, however, the broken line showing the display range PN is notdisplayed and not visually recognized by the user.

If the type of the marker MK1 as the marker is specified (step S16 inFIG. 4), the marker determination part 165 determines whether or not themarker MK1 is a marker related to an operation instruction stored in thehead mounted display device 100 in advance (step S18). In theembodiment, if the related markers are detected along the sequence ofthe operation instructions stored in advance, the marker determinationpart 165 displays images corresponding to the detected markers.Accordingly, if the detected marker is not related to any operationinstruction or related to an operation instruction, but not along thesequence of the operation instructions, the part determines that thedetected marker is a marker not related to the operation instruction.Further, if the area image AI contains a plurality of images of markers,the marker determination part 165 detects only the related marker alongthe sequence of the operation instructions. In the processing at stepS18, if a determination that the marker MK1 is not a marker related tothe stored operation instruction is made (step S16: NO), the markerdetermination part 165 detects an image of a marker different from themarker MK1 again (step S14).

In the processing at step S18, if a determination that the marker MK1 isa marker related to the operation instruction is made (step S18: YES),the marker determination part 165 specifies the shape of the marker MK1,and thereby, specifies a positional relationship between the age displayunit 20 and the marker MK1 placed in the distribution center 300 (stepS20). Then, the marker determination part 165 allows the image displayunit 20 to display an instruction image of the operation instructioncorresponding to the specified positional relationship (step S22 in FIG.4).

FIG. 8 is an explanatory diagram showing a visual range VR of the userwhen an instruction image IM1 is displayed at the start point ST. FIG. 8shows the visual range VR visually recognized by the user when theinstruction image IM1 corresponding to the marker MK1 is displayed andthe user at the start point ST faces in the direction DR1. In this case,unlike the visual range VR shown in FIG. 7, the user visually recognizesthe instruction image IM1. The marker determination part 165 comparesshapes of outer frames between image data of the marker MK1 stored inadvance and the image of the imaged marker MK1. As shown in FIGS. 5 and8, the outer frames of the image data of the marker MK1 and the image ofthe imaged marker MK1 are square. Accordingly, the marker determinationpart 165 specifies an existence of the marker MK1 in front of the camera61 as a positional relationship of the image display unit 20 with themarker MK1. Further, the marker determination part 165 compares sizesbetween the image data of the marker MK1 and the image of the imagedmarker MK1, and thereby, specifies a distance between the image displayunit 20 and the marker MK1 placed in the distribution center 300 as apositional relationship between the image display unit 20 and the markerMK1. Then, the marker determination part 165 allows the image displayunit 20 to display the instruction image IM1 stored in advance inresponse to the specified positional relationship. If the user at thestart point ST faces in the direction DR1 (FIG. 6), the markerdetermination part 165 displays a character image of “TURN TO LEFT” andan arrow pointing to the left as the instruction image IM1 to bevisually recognized in a location at the same distance with that to themarker MK1. The instruction image IM1 and an instruction image IM2, aninstruction image IM3, which will be described later, correspond to aspecific image in the appended claims.

If the instruction image IM1 is displayed (step S22 in FIG. 4), themarker determination part 165 monitors a change to a non-detection statein which the detected marker MK1 is no longer detected by changes inlocation and orientation of the user (step S24). If the marker MK1 iscontinuously detected (step S24: NO), the marker determination part 165determines whether or not the positional relationship between the imagedisplay unit 20 and the marker MK1 placed in the distribution center 300has changed (step S26). For example, if the user moves to a locationcloser to the marker MK1 along the direction DR1 than the start point STwhile facing in the direction DR1 and a determination that thepositional relationship between the image display unit 20 and the markerMK1 has changed is made (step S26: YES), the marker determination part165 specifies the positional relationship between the image display unit20 and the marker MK1 (step S20). As the specified positionalrelationship, if the distance between the image display unit 20 and themarker MK1 is smaller than a predetermined value, the markerdetermination part 165 hides the image of the instruction image IM1 andnewly displays a character image of “RETURN” as another instructionimage stored in advance (step S22). The marker determination part 165displays “RETURN” as the new character image, and thereby, prompts theuser to move backward. If the user passes within the distribution center300 toward the shelf SH according to the newly displayed character imageand the state has changed from the detection state to the non-detectionstate of the marker (step S24: YES), the marker determination part 165hides the instruction image. Then, again, the marker determination part165 monitors detection of a marker within the area image AI (step S14).

Next, the case where the processing from step S14 to step S24 isrepeated, and thereby, the user at the relay point LC shown in FIG. 6faces in the direction DR2 is explained. FIG. 9 is an explanatorydiagram showing a visual range VR of the user when the instruction imageIM2 is displayed at the relay point LC. FIG. 9 shows the visual range VRvisually recognized by the user when the instruction image IM2corresponding to the marker MK2 is displayed and the user at the relaypoint LC faces in the direction DR2. The instruction image IM2 is aninstruction image displayed in response to the image display unit 20 andthe marker MK2 specified by the marker determination part 165. As shownin FIG. 9, the instruction image IM2 includes an image of an arrowpointing to the shelf SH on which the component to be acquired by theuser is stored and a character image of “THIS SHELF”. The markerdetermination part 165 displays the instruction image IM2 to be visuallyrecognized in a location at the same distance with that to the markerMK2. The user is guided to the shelf SH of the distribution center 300by the instruction images including the instruction image IM2 and theinstruction image IM1.

FIG. 10 is an explanatory diagram showing a visual range VR of the userin front of the shelf SH. As shown in FIG. 10, the user visuallyrecognizes a plurality of bolts BO stored in a case CA placed on theshelf SH, a marker MK3, and the instruction image IM3. The instructionimage IM3 is an image displayed in response to a positional relationshipbetween the image display unit 20 and the marker MK3 specified based onan image of the imaged marker MK3 and image data of the marker MK3stored in advance. The marker determination part 165 displays acharacter image of “TAKE THREE BOLTS” as the instruction image IM3. Themarker determination part 165 displays the instruction image IM3 to bevisually recognized in a location at the same distance with that to themarker MK3.

In the processing at step S22 in FIG. 4, if the instruction image IM3 isdisplayed and the marker MK3 is continuously detected (step S24: YES),the marker determination part 165 monitors a change of the positionalrelationship between the image display unit 20 and the marker MK3 (stepS26). If the positional relationship does not change (step S26: NO), themarker determination part 165 monitors a lapse of a predetermined periodof time in the state in which the instruction image IM3 is displayed(step S28) If the predetermined period of time has elapsed (step S28:YES), the marker determination part 165 hides the instruction image IM3(step 330). In the embodiment, the instruction image IM3 is hidden afterthe lapse of the predetermined period of time, and thereby, the user mayvisually recognize the wider outside scenery SC. If the predeterminedperiod of time has not elapsed in the processing at step 328 (step S28:NO) or if the instruction image IM3 is hidden (step S30), the markerdetermination part 165 determines whether or not to end the instructionimage display processing (step S32). If the instruction image displayprocessing is not ended (step S32: NO), the processing at step 326 andsubsequent steps is performed. If the operation part 135 receives apredetermined operation to end the instruction image display processingor the operation instruction stored in advance is ended, the markerdetermination part 165 ends the instruction image display processing(step S32: YES).

As described above, in the head mounted display device 100 in theembodiment, the camera 61 images an outside scenery and acquires animaged image and, if the imaged image contains a marker, the markerdetermination part 165 displays an image corresponding to a combinationof the type of the detected marker and the shape of the image in thedisplay range PN of the image display unit 20. Accordingly, in the headmounted display device 100 of the embodiment, an image corresponding toa positional relationship between the user and the detected markerspecified in response to the shape of the detected marker is displayed,and thereby, the user may visually recognize optimal instruction imagesin response to the types and the positional relationships of thedetected markers and efficiently perform a picking operation or thelike.

Further, in the head mounted display device 100 of the embodiment, themarker determination part 165 compares the sizes between image data ofthe marker MK1 and the image of the detected marker MK1, and thereby,specifies the distance between the image display unit 20 and the markerMK1 placed in the distribution center 300 as the positional relationshipbetween the image display unit 20 and the marker MK1. The markerdetermination part 165 displays the image corresponding to thecombination of the type of the specified marker and the size of theimage in the display range PN. Accordingly, in the head mounted displaydevice 100 of the embodiment, even when the same marker MK1 is imaged, aplurality of images may be visually recognized by the user in responseto the specified distances, and even one marker may provide moreinformation suitable for the positional relationship with the user tothe user.

Furthermore, in the head mounted display device 100 of the embodiment,the marker determination part 165 sets the distances from the opticalimage display parts 26, 28 to the location where the virtual image isformed to be equal to the distance from the camera 61 to the imagedmarker. Accordingly, in the head mounted display device 100 of theembodiment, the virtual image may be formed in the location where theuser easily and visually recognizes the virtual image, and the imagedisplayed in the display range PN may be promptly and visuallyrecognized by the user.

Further, in the head mounted display device 100 of the embodiment, ifthe related markers are detected along the sequence of the flow of theoperation instructions stored in advance, the marker determination part165 displays the image corresponding to the detected markers.Accordingly, in the head mounted display device 100 of the embodiment,only the related markers along the sequence of the operationinstructions are detected from the plurality of markers placed in thedistribution center 300, and thereby, erroneous operation instructionsin a different sequence or the like may be reduced and efficientoperations be performed by the user.

Furthermore, in the head mounted display device 100 of the embodiment,the marker determination part 165 determines whether or not there is animage of a marker stored in the memory part 120 in the rangecorresponding to the inside of the area image in the imaged image of thecamera 61. If the imaged image contains the image of the marker, but theimage of the entire of the marker is not contained within the areaimage, the ma determination part 165 determines that the imaged imagedoes not contain the image of the marker. Moreover, in the head mounteddisplay device 100 of the embodiment, if a marker is continuously imagedwithin the area image in a predetermined period of time, the markerdetermination part 165 detects the marker as a marker, and does notdetect a marker instantaneously contained within the area image as amarker. Accordingly, in the head mounted display device 100 of theembodiment, a marker only contained in the imaged image is not detectedas a marker for displaying a corresponding image, and the load of theprocessing on the head mounted display device 100 may be reduced. Inaddition, in the head mounted display device 100 of the embodiment,detection of markers not intended by the user may be suppressed and theuser may be prompted by operation instructions with higher accuracy.

Further, in the head mounted display device 100 of the embodiment, ifthe area image AI in an imaging range contains a plurality of images ofmarkers, the marker determination part 165 detects only the relatedmarkers along the sequence of the operation instructions stored inadvance. Accordingly, in the head mounted display device 100 of theembodiment, images corresponding to the markers suitable for theoperation instructions stored in advance among the plurality of markersin the imaging range are automatically displayed, and thereby, erroneousoperation instructions for the user may be reduced and efficientoperations may be performed by the user.

Furthermore, in the head mounted display device 100 of the embodiment,in the marker MK1, the image formed in the square includes white areaand a black area, and thereby, the detected marker MK1 is not complexand control for displaying a corresponding image by the detection of themarker MK1 may be performed more easily. In addition, the marker MK1 isformed in black and white only, and thereby, control with less erroneousdetection may be performed.

B. Modified Examples:

The invention is not limited to the above described embodiment, but maybe implemented in various forms without departing from the scopethereof. The following modifications may be made, for example.

B1. Modified Example 1:

In the above described embodiment, the case where the marker placed inthe distribution center 300 is located in front of the user and theshape of the outer frame of the marker is detected as square isexplained as an example, however, the marker is not necessarily locatedin front of the user. For example, when a marker is detected diagonallyahead of the camera 61, an image prompting the user to change theorientation is displayed, and the user changes the orientation and islocated in front of the marker, an image of an operation instruction maybe displayed. Further, a setting of not detecting a marker not locatedin front of the user may be made.

In the above described embodiment, the instruction image correspondingto the detected marker is once displayed and then hidden in theprocessing at step S25 and step S32 in FIG. 4, however, the instructionimage is not necessarily hidden. For example, in the instruction imagedisplay processing in the embodiment, the instruction images displayedalong the sequence of the operation instructions may not be hidden, butthe reduced instruction images may be left in the corner of the displayrange PN. In the modified example, the instruction images that have beenalready displayed may be constantly and visually recognized by the user,and errors in the operation instructions for the user may be reduced.Further, when the instruction image is hidden, a number of a countdownof seconds to hiding of the image may be additionally displayed.Furthermore, switching between display and non-display of theinstruction image and the time to non-display of the displayedinstruction image may be set by the user.

In the above described embodiment, the distance from the image displayunit 20 to the location where the virtual image is formed may be set tobe equal to the distance from the camera 61 to the detected marker,however, the setting of the distance from the image display unit 20 tothe location where the virtual image is formed may be variouslymodified. For example, regardless of a marker to be detected, a locationwhere a virtual image corresponding to the marker is formed may be setto a location at infinity from the image display unit 20. Further, thelocation where the virtual image is formed may be freely set by the useroperating the operation part 135. Furthermore, the line-of-sightdirection of the user may be imaged by a camera different from thecamera 61, and the location where the virtual image corresponding to themarker is formed may be set based on the angle of convergence of theline of sight of the user specified based on the line-of-sightdirection. In addition, the distance between the image display unit 20and the user is not specified, but only a positional relationship withrespect to the orientations between the marker and the line-of-sightdirection of the user may be specified by the type of the detectedmarker and the shape of the outer frame of the image of the marker, andthe image corresponding to the detected marker may be displayed. Thedistance from the image display unit 20 to the marker and the positionalrelationship with respect to the orientations between the marker and theline-of-sight direction of the user may be set by the user.

In the above described embodiment, if a marker not along the sequence ofthe operation instructions detected, an image corresponding to thedetected marker is not displayed, however, a setting of display andnon-display of the image corresponding to the detected marker may bevariously modified. For example, a marker related to an operationinstruction and a marker not related may not be distinguished and animage corresponding to the detected marker may be displayed regardlessof the contents of the operation instructions stored in advance. In themodified example, an image corresponding to a marker may be visuallyrecognized by the user even when operation instructions are not storedin advance in the memory part 120. Further, in the instruction imagedisplay processing, the determination by the marker determination part165 as to whether or not the marker is a marker related to an operationinstruction is not necessarily made, and the load of the processing onthe head mounted display device 100 may be reduced. Furthermore, amarker for which a corresponding image is displayed may be set accordingto another condition than the sequence of the operation instructions Forexample, the detected marker may be a marker for displaying acorresponding image only to a user of the head mounted display device100 who pays a fee in advance.

Further, in the above described embodiment, the distribution center 300in which operation instructions are given to the user by display of theimages corresponding to the markers is explained as an example, however,the system of displaying images using markers is not limited to that,but may be variously modified. For example, in a museum, a route forviewing paintings and sculptures may be predetermined and markers placedin the path may be detected, and thereby, the route of the user may beshown. These systems may be used not in the museums, but in leisurefacilities including amusement parks and shopping malls.

In the above described embodiment, as the images of the operationinstructions corresponding to the images of the markers and displayed inthe display range PN, the instruction images IM1 of “TURN TO LEFT” and“RETURN” corresponding to the markers MK1 are explained as examples,however, the images corresponding to the images of the imaged markersmay be variously modified. For example, when an image of a marker placedin a T-junction in the route in the museum and prompting the user of thehead mounted display device 100 to take the route to the right isimaged, an image displayed in the image display maximum range PN mayvary depending on the location of the image of the marker in a detectionrange in which the image of the marker is detected and the shape of theouter frame of the image of the marker. In this case, when the image ofthe marker is detected on the right in the detection range and the outerframe of the image of the marker of the square outer frame is detectedas a trapezoidal image, the user seeks to turn the T-junction to theleft, and “WRONG WAY” as a character image corresponding to the image ofthe marker may be displayed in the display range PN. Conversely, whenthe image of the marker is detected on the left in the detection rangeand the outer frame of the image of the marker is detected as atrapezoidal image, the user seeks to turn the T-junction to the right,and “RIGHT WAY” as a character image may be displayed in the displayrange PN.

B2 Modified Example 2:

In the above described embodiment, the marker is detected when the areaimage AI contains the image of the entire marker, however, the method ofdetecting the marker may be variously modified. For example, the areaimage AI may not be displayed, but the marker contained in the imagedimage imaged by the camera 61 may be detected. Or, after the marker iscontained in the imaged image, the area image AI may be displayed.Further, the shape of the range within the area image AI may be set bythe user.

If an imaged image contains a plurality of markers, the markerdetermination part 165 may detect a marker whose outer frame has themaximum size and display the image corresponding to the maximum marker,but may not display images corresponding to the other markers. In themodified example, the image of the maximum marker is automaticallydetected from the plurality of markers, and thereby, the marker closestto the user may be automatically detected and convenience of the usermay be improved.

FIGS. 11 to 13 are explanatory diagrams showing examples of the markersin modified examples. FIG. 11 shows the example of a barcode detected asa marker. FIG. 12 shows the example of a QR code (registered trademark)detected as a marker. FIG. 13 shows the example of an ID marker detectedas a marker. The marker determination part 165 may detect the markersshown in FIGS. 11 to 13 or detect markers uniquely created with respectto each distribution center.

Further, in the above described embodiment, the marker MK1 is an imageincluding a white area and a black area in a square, however, the markeris not limited to that, but may be variously modified. For example, theouter frame of the marker may be a rectangular shape or a circularshape. Or, as the marker, other colors than white or black (e.g., red)may be used. In the modified example, a marker using other colors thanwhite or black is created, and thereby, even a smaller marker mayprovide more information to the user compared to the marker formed inwhite and black only.

B3. Modified Example 3:

In the above described embodiment, the operation part 135 is formed inthe control unit 10, however, the form of the operation part 135 may bevariously modified. For example, a user interface as the operation part135 may be provided separately from the control unit 10. In this case,the operation part 135 is separated from the control unit 10 with thepower source 130 etc. formed therein, and the part may be downsized andthe operability of the user is improved. Further, the camera 61 isformed in the image display unit 20, however, the camera 61 may beprovided separately from the image display unit 20 and image outsidesceneries.

For example, the image light generation part may include an organic EL(Organic Electro-Luminescence) display and an organic EL control unit.Further, for example, for the image generation part, in place of theLCD, an LCOS (Liquid crystal on silicon, LCOS is a registeredtrademark), a digital micros or device, or the like may be used.Furthermore, SW example, the invention may be applied to a laser retinaprojection type head mounted display.

Further, for example, the head mounted display device 100 may have aform having an optical image display part that covers only a part of theeye of the user, in other words, a form of an optical image display partthat does not completely cover the eye of the user. Furthermore, thehead mounted display device 100 may be the so-called monocular-type headmounted display. In addition, the head mounted display device 100 is thebinocular-type optically-transmissive head mounted display, however, theinvention may be similarly applied to a head mounted display device inother forms including a video-transmissive type, for example.

Further, ear-fit-type or headband-type earphones may be employed or theearphones may be omitted. Furthermore, the head mounted display devicemay be formed as a head mounted display device mounted on a vehicle ofan automobile, an airplane, or the like, for example. In addition, forexample, the head mounted display device may be formed as a head mounteddisplay device build in a body protector including a hardhat

B4 Modified Example 4:

The configuration of the head mounted display device 100 in theembodiment just an example and may be variously modified. For example,the arrow key 16 provided in the control unit 10 may be omitted oranother operation interface such as an operation stick may be providedin addition to the arrow key 16 and the track pad 14 or in place of thearrow key 16 and the track pad 14. Further, the control unit 10 may havea configuration to which an input device such as a keyboard or mouse canbe connected and receive input from the keyboard or the mouse.

Furthermore, as the image display unit, in place of the image displayunit 20 worn like spectacles, an image display unit of another systemsuch as an image display unit worn like a hat may be employed, forexample. Further, the earphones 32, 34 may be appropriately omitted.Furthermore, in the above described embodiment, the LCD and the lightsource are used as the configuration of generating image light, however,in place of them, another display device such as an organic EL displaymay be employed.

FIGS. 14A and 14B are explanatory diagrams showing outer configurationsof head mounted display devices in modified examples. In the case of theexample of FIG. 14A, the difference from the head mounted display device100 shown in FIG. 1 is that an image display unit 20 a includes a rightoptical image display part 26 a in place of the right optical imagedisplay part 26 and a left optical image display part 28 a in place ofthe left optical image display part 28. The right optical image displaypart 26 a is formed to be smaller than the optical members of the abovedescribed embodiment, and provided in the obliquely upper part of theright eye of the user when a head mounted display device 100 a is worn.Similarly, the left optical image display part 28 b is formed to besmaller than the optical members of the above described embodiment, andprovided in the obliquely upper part of the left eye of the user whenthe head mounted display device 100 a is worn. In the case of theexample of FIG. 14B, the difference from the head mounted display device100 shown in FIG. 1 is that an image display unit 20 b includes a rightoptical image display part 26 b in place of the right optical imagedisplay part 26 and a left optical image display part 28 b in place ofthe left optical image display part 28. The right optical image displaypart 26 b is formed to be smaller than the optical members of the abovedescribed embodiment, and provided in the obliquely lower part of theright eye of the user when the head mounted display is worn. The leftoptical image display part 26 b is formed to be smaller than the opticalmembers of the above described embodiment, and provided in the obliquelylower part of the left eye of the user when the head mounted display isworn. As described above, it is only necessary that the optical imagedisplay unit is provided near the eve of the user. Further, the sizes ofthe optical members forming the optical image display unit may bearbitrary, and the head mounted display device 100 in which the opticalimage display unit covers only a part of the eye of the user, in otherwords, the optical image display unit does not completely cover the eyeof the user may be implemented.

Further, in the above described embodiment, the head mounted displaydevice 100 may guide image lights representing the same image to theleft and right eyes of the user and allows the user to visuallyrecognize a two-dimensional image, or may guide image lightsrepresenting different images to the left and right eyes of the user andallows the user to visually recognize a three-dimensional image.

Furthermore, in the above described embodiment, part of theconfiguration implemented by hardware may be replaced by software, or,conversely, a part of the configuration implemented by software may bereplaced by hardware. For example, in the above described embodiment,the image processing part 160 and the sound processing part 170 may beimplemented by the CPU 140 reading out and executing computer programs,however, these functional parts may be implemented by a hardwarecircuit.

In addition, in the case where part or all of the functions of theinvention are implemented by software, the software (computer programs)may be stored and provided in computer-readable media. In the invention,“computer-readable media” include not only portable recording media suchas a flexible disk or a CD-ROM but also internal memory devices withinthe computer such as various RAMs and ROMs and external memory devicesfixed to the computer such as a hard disk.

Further, in the above described embodiment, as shown in FIGS. 1 and thecontrol unit 10 and the image display unit 20 are formed as separateconfigurations, however, the configurations of the control unit 10 andthe image display unit 20 are not limited to those, but may be variouslymodified. For example, all or part of the configurations formed in thecontrol unit 10 may be formed inside of the image display unit 20.Further, the power source 130 in the embodiments may be singly formedand replaceable, or the configuration formed in the control unit 10 maybe redundantly formed in the image display unit 20. For example, the CPU140 shown in FIG. 2 may be formed in both the control unit 10 and theimage display unit 20, or the functions performed by the CPU 140 formedin the control unit 10 and the CPU formed in the image display unit 20may be separated.

The invention is not limited to the above described embodiments andmodified examples, but may be implemented in various configurationswithout departing from the scope thereof. For example, the technicalfeatures in the embodiments and the modified examples corresponding tothe technical features in the respective forms described in “SUMMARY”may be appropriately replaced or combined in order to solve part or allof the above described problems or achieve part or all of the abovedescribed advantages. Further, the technical features may beappropriately deleted unless they are described as essential features inthe specification.

What is claimed is:
 1. A display device comprising: an image displaythat displays an image light and enables the user to visually recognizean outside scenery; an image sensor that images the outside scenery; anda controller that, when an imaged image contains a specific mark,controls the image display to display a specific image light associatedwith the specific mark, and that determines whether the specific markcontinues to be detected for a predetermined time, wherein when thespecific image light is displayed and the specific marker iscontinuously detected for the predetermined time, the controllermonitors a change of a positional relationship between the displaydevice and the specific mark, if the positional relationship has notchanged, the controller determines whether the specific image light hasbeen displayed for a predetermined period of time, and if thepredetermined period of time has elapsed, the controller hides thespecific image light.
 2. The display device according to claim 1,wherein the specific image light is associated with a combination of atype of the specific mark and a size of the specific mark.
 3. Thedisplay device according to claim 2, wherein the controller sets adistance from the image display to a location where the specific imagelight is formed to be equal to a distance from the image display to thespecific mark.
 4. The head mounted display device according to claim 1,wherein the specific image light is associated with a combination of anorder of imaging of the one specific mark contained in each of aplurality of imaged images, the type of the specific mark, and the shapeof the specific mark.
 5. The head mounted display device according toclaim 1, wherein, when the imaged image contains a plurality of marks,the controller controls the image display to display the specific imagelight associated with the combination of one mark of the plurality ofmarks which has the maximum size of the plurality of marks.